perf(retrieval): add content trigram index - #389
Merged
Conversation
…ine collection, and fallback auto-collect (#125) * feat: enhance navigation state with error handling, custom tool parameters, and improved workflow plan caching * feat(retrieval): enhance agentic navigation with scope tracking, outline collection, and fallback auto-collect - Replace scope_stack with flat current_scope for cleaner BACK/DRILL navigation - Add structured exit_reason tracking across all navigation exit paths - Implement fallback auto-collect when navigation interrupted by budget/latency/max_steps - Fix outline collections to not exclude children from future navigation - Add BACK target validation with path-parent fallback - Fix evidence tree indentation for nested sections - Fix pyright type errors in plan_service by using explicit kwargs - Fix lint: remove extraneous f-string prefixes in asset tools
* feat: inject discovery signals as advisory hints into agent document selection prompts * fix(agentic): resolve evidence tree orphan nesting, outline leakage, and enforce English agent reasoning - Fix _build_outline_subtree to reparent existing children under newly created parent nodes, preventing orphan depth errors (Bug 1) - Filter outline_items in _hydrate_collected to exclude ancestor/sibling context from load_child_sections, eliminating echo structures (Bug 1) - Change render_evidence guard from has_content() to has_leaf_content(), skipping documents with outline-only trees when no chunks were collected (Bug 2) - Add English language requirement to COLLECTOR_PROMPT and _PLANNER_PROMPT for agent-generated free-text fields like reason and reasoning_summary (Bug 3) - Remove unused imports flagged by ruff lint Closes #117 * refactor: remove inspect tool, update asset search with optional VLM, and improve navigation stop/fallback logic * feat: implement H2-aware shard refinement logic to improve document splitting granularity and adherence to size constraints. * test(contract): update discovery selection contract tests to match id_to_path signature
…actor (#128) - Remove image_topic_hints / table_topic_hints params from build_asset_tools_block (hints were never used in the new prompt format) - Remove dead load_asset_topic_hints function from assets.py - Remove hint-loading block and wasted DB queries from navigate_step in tools.py
…-staging Route oversized PDF shards through S3 for MinerU
…model upgrade (#133) - Add Phase 2.5 discovery fallback for unnavigated documents that have bottom-discovery hits but were missed by KG document selection - Add adjust_budget_snapshot() to pre-adjust budget state shown to LLM - Update prompt_service with model-specific formatting (DeepSeek/Qwen) - Upgrade IMAGE_MODEL to qwen2.5-vl-72b-instruct for better VLM quality - Remove unused DocTreeNode import (lint fix) - Clean up navigation tools dead code Closes #132
* refactor: remove legacy retrieval routes and standardize agentic orchestration with enhanced decision tracing and token tracking * feat: implement structured DecisionTraceStep tracking and refine agentic navigation prompt constraints * feat(agentic): reconcile navigation asset search flow
* fix: route oversized pdf shards through s3 * Revert "Route oversized PDF shards through S3 for MinerU" * ci: move production release flow to tags * docs: detail contribution release workflow * ci: skip deployment image builds on pull requests * ci: trigger production deploy from releases * docs: hide internal environment branch from public flow * docs: remove release flow from public docs --------- Co-authored-by: suguanYang <wangbinqi77@gmail.com> Co-authored-by: suguanYang <suguanyang@ylioo.com>
…staging-20260617 chore: sync main into staging
research demos
introduce permission
optimize Document apis
document pagination
base image
Fix retrivel response
Staging page memory fixing
Fixing page citation result harden
Bug fix and new BYOK
chore: sync main into staging (agentic per-doc cap + doc-agent)
The bounded gevent cancellation raced with an active PyMuPDF subprocess, removed its workspace, and still missed the ECS stop deadline. Keep the independent Redis visibility watchdog as the accepted interruption fallback.
…-shutdown-staging revert(worker): rely on visibility-timeout recovery (staging)
fix(ecs): preserve validated staging worker CPU
…grep perf(retrieval): add content trigram index for regex grep
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| from alembic import op | ||
|
|
||
|
|
||
| revision: str = "0a1b2c3d4e5f" |
|
|
||
|
|
||
| revision: str = "0a1b2c3d4e5f" | ||
| down_revision: str | None = "9f0a1b2c3d4e" |
|
|
||
| revision: str = "0a1b2c3d4e5f" | ||
| down_revision: str | None = "9f0a1b2c3d4e" | ||
| branch_labels: Sequence[str] | None = None |
| revision: str = "0a1b2c3d4e5f" | ||
| down_revision: str | None = "9f0a1b2c3d4e" | ||
| branch_labels: Sequence[str] | None = None | ||
| depends_on: Sequence[str] | None = None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the staging-only content trigram migration into
mainand resolves the Alembic heads introduced by the parallel retrieval-index migrations.pg_trgmGIN index ondocument_chunks.content.d3e4f5a6b7c8so Alembic has one head after both index branches.This replaces the incorrectly targeted PR #388, which was closed because the repository workflow uses
mainas the base branch.Validation:
cd apps/api && uv run python -m pytest tests/migrations/test_schema_contract.py -q(16 passed);uv run --all-packages --group lint ruff check apps packagespassed.